R-TF-028-004 Data Annotation Instructions - Visual Signs
Table of contents
Context
The Legit.Health Plus device is intended to provide quantifiable data on the intensity, count, and extent of a wide range of visible clinical signs[cite: 551]. To develop and validate the AI/ML algorithms that perform this quantification, a high-quality, ground-truth dataset is required.
This document provides the instructions for medical experts on how to annotate dermatological images for these specific visual signs. The annotation process is distinct from the ICD-11 and Binary Indicator labeling and involves several different tasks, including assigning intensity scores, drawing bounding boxes for counting, and creating polygons to define extent. This process is fundamental for training the supervised learning models.
Objectives
- To create a comprehensive, multi-modal ground-truth dataset for the quantification of visual clinical signs.
- To provide clear, unambiguous instructions to medical annotators to ensure consistency and accuracy across all annotation tasks.
- To establish a robust foundation for training, validating, and testing the AI/ML models responsible for quantifying intensity, count, and extent.
Annotation Personnel
Role
Medical expert (e.g., Dermatologist, Wound Care Specialist).
Qualifications
- Required: Board-certified specialist relevant to the pathology being assessed (e.g., Dermatology, Plastic Surgery).
- Experience: A minimum of three years of clinical experience in assessing and managing relevant skin conditions is required to ensure expertise.
Responsibilities
- To review and understand these annotation instructions thoroughly.
- To perform annotations on the provided images using the designated annotation platform.
- To apply clinical expertise to complete each annotation task accurately and consistently according to the specific guidelines below.
General Annotation Workflow
All annotations will be performed using a web-based medical image annotation platform. For each image assigned, the annotator will:
- Examine the image to identify all relevant clinical signs present.
- For each sign identified, select the corresponding annotation task from the tool's interface.
- Perform the annotation as per the specific instructions for that task type (Intensity, Count, or Extent).
- Repeat the process until all visible and relevant signs in the image have been annotated.
Specific Annotation Instructions by Task Type
This section details the distinct annotation methodologies for each category of visual sign.
Task: Intensity Annotation (Ordinal & Categorical)
This task is for signs where severity is assessed on a scale or by category.
- Signs to Annotate:
- Ordinal Scale (e.g., 0-4): Erythema, Desquamation, Induration, Crusting, Xerosis (dryness), Swelling (oedema), Oozing, Excoriation, Lichenification.
- Categorical (Non-Ordinal): Necrotic Tissue, Granulation Tissue, Epithelialization, Slough or Biofilm, Exudation.
- Instruction: For each sign present in the image, select the numerical value or descriptive category from the provided list that best represents its intensity or type.
- Guidelines & Example (Ordinal - Erythema): The platform will present a scale. Detailed clinical scoring guides based on established literature will be provided as a reference.
| Score | Description |
|---|---|
| 0 | None |
| 1 | Mild, faint redness |
| 2 | Moderate, clear redness |
| 3 | Severe, deep redness |
| 4 | Very severe, fiery redness |
Task: Count Annotation (Bounding Boxes)
This task is for discrete, countable lesions.
- Signs to Annotate: Nodule, Papule, Pustule, Cyst, Comedone, Abscess, Draining Tunnel, Inflammatory Lesion.
- Instruction: Using the bounding box tool, draw a tight rectangle around each individual instance of the specified sign.
- Guidelines:
- The box must contain the entire visible lesion with as little background as possible.
- Overlapping lesions should be boxed individually if they are clinically distinguishable as separate entities.
- Ensure every instance of the target sign within the image is annotated.
Task: Extent Annotation (Polygons)
This task is for signs that cover a surface area.
- Signs to Annotate: Hair Loss, Maceration, Hypopigmentation or Depigmentation, Hyperpigmentation, Scar, Wound Border, Undermining, Exposed Wound.
- (Note: Some signs like Erythema may also be annotated for extent in specific contexts, which will be clearly indicated in the task instructions).
- Instruction: Using the polygon tool, carefully trace the complete boundary of the area affected by the specified sign.
- Guidelines:
- The outline should be as precise as possible, closely following the visible edge of the sign.
- Place polygon points close enough together to accurately capture curves.
- For conditions with multiple, non-contiguous patches, draw a separate polygon for each distinct area.
Quality Control
To ensure the creation of a high-fidelity ground truth and to measure inter-observer variability, a rigorous quality control process will be followed.
- Multi-Annotator Process: To ensure the highest quality, images will be annotated by one or more qualified medical experts. Based on expert opinion regarding the complexity and subjectivity of each specific annotation task (e.g., intensity of erythema vs. counting nodules), a lead medical expert will determine the required number of annotators to establish a reliable consensus.
- Consensus Ground Truth: When multiple experts annotate the same image, the final ground-truth label will be established by pooling their independent assessments. The method for establishing consensus will depend on the task (e.g., mean/median for intensity scores, voting for categories, or algorithmic fusion for bounding boxes/polygons).
- Senior Review: A designated senior specialist will review cases with high inter-annotator disagreement to resolve discrepancies and ensure the final label is clinically sound.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001